Repository navigation
kubectl ate: authenticate with ExecCredential plugins named by --credential-plugin - #2851
Max Thompson (thompsonmax) wants to merge 2 commits into
Conversation
f8db639 to
cb06553
Compare
cb06553 to
56c6bf0
Compare
56c6bf0 to
9cb63fd
Compare
kubectl ate minted one hour-long token for the shared ate-client ServiceAccount when it connected and never refreshed it, so long-running clients failed after an hour. Every user also shared that one identity, and using another identity provider meant piping a raw JWT into --token-file on every command. --credential-plugin, or $ATE_CREDENTIAL_PLUGIN, names a kubectl credential plugin. A value that names an existing file runs with no arguments; any other value is split into a command and arguments, honoring shell quoting. kubectl ate builds the exec stanza in memory (client.authentication.k8s.io/v1, interactiveMode IfAvailable, provideClusterInfo) and passes its --kubeconfig and --context to the plugin as $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT. Passing both --token-file and --credential-plugin is an error. ateclient runs the plugin through client-go's exec authenticator, caches the credential until its expirationTimestamp, and retries an RPC once with a fresh credential when ate-api returns Unauthenticated. If the plugin returns the token that was just rejected, the client runs it again with ATE_REFRESH_CREDENTIAL=1 so that a plugin with a cache returns a new one. The ate-client token minted when there is no plugin or token file is now refreshed a minute before it expires. A credential source that fails locally is reported as a local error rather than an Unauthenticated status. kubectl ate credential is a built-in plugin. It mints a token for ate-client, or for a ServiceAccount named with --service-account, and caches it on disk until shortly before it expires; --refresh mints a new one. -o token prints only the token, and credential wrap turns any command that prints a JWT into a plugin. kubectl ate whoami calls a new Authentication.WhoAmI RPC that returns the principal ate-api authenticated the caller as. It is a separate service because it operates on no resource. The docs cover the flag, a per-user ServiceAccount recipe with the RBAC a non-admin needs, what deleting a ServiceAccount does not revoke, and in-cluster workloads.
The Unauthenticated retry also covered the minted ate-client token. A minted token is replaced before it expires, so when ateapi rejected it, for example because it does not trust the issuer, every failing RPC was sent twice and minted a new token. Only exec plugin credentials are now retried, and the token-file, stdin, and minted credentials lose their refresh methods. All of them are still fetched before each RPC, so a credential that cannot be fetched is still reported as a local error. RPCs rejected together each forced a plugin rerun, because one refresh could charge its 401 against the token another had just fetched. Refresh is now serialized, so they share one refresh. client-go cleans a plugin's command, which turned ./plugin into a bare name looked up on PATH, so kubectl ate now makes a command containing a separator absolute. It also always sets ATE_KUBECONFIG and ATE_KUBE_CONTEXT, empty when the flag is not given, so the plugin no longer inherits values from the user's environment that kubectl ate itself ignores. The e2e authz suite now checks the principal WhoAmI reports for a ServiceAccount token sent from a token file and from a credential plugin, minting it with ateclient.MintServiceAccountToken. The test plugin claims its run number with an exclusive create so that concurrent runs are counted, the concurrent refresh test runs readers alongside the refreshers, the missing-plugin test no longer depends on client-go's rate-limited install hint, and the credential command tests use the package's existing runRootCLI helper.
9cb63fd to
a2892f6
Compare
| if c.token == "" || !c.now().Before(c.expiry.Add(-tokenExpiryMargin)) { | ||
| token, expiry, err := c.mint(ctx) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| c.token, c.expiry = token, expiry | ||
| } |
There was a problem hiding this comment.
🤖 nit 🟢 – When re-minting fails inside the one-minute margin, the RPC fails even though the cached token is still valid. Consider returning the current token while now < expiry and only surfacing the mint error once it has actually expired, so a transient TokenRequest failure does not take down a long-lived client early.
| } | ||
| // Keyed by the kubeconfig too: whoever can mint here is not necessarily | ||
| // whoever can mint through another kubeconfig context on the same host. | ||
| key := []string{config.Host, kubeconfigPath, kubeContext, namespace, name, audience} |
There was a problem hiding this comment.
🤖 nit 🟢 – With --kubeconfig and --context unset, this key does not distinguish different $KUBECONFIG values or a changed current-context that resolve to the same config.Host, which is the case the comment above is guarding against. Keying on the resolved context name and its AuthInfo name from RawConfig() would tie the entry to the minting identity rather than to the paths.
| Google service account ID tokens or GitHub Actions OIDC tokens: any service | ||
| account or workflow can request one for any audience. | ||
|
|
||
| `kubectl ate credential` is the built-in plugin. It mints a ServiceAccount |
There was a problem hiding this comment.
nit: in some places it says "kubectl ate credential" in others "kubectl-ate credential".
Benjamin Elder (BenTheElder)
left a comment
There was a problem hiding this comment.
AI-assisted first pass, inline comments are mostly small. Two bigger comments:
-
Can we split this up? WhoAmI and
kubectl ate credential/wrapcould each be their own PR, this is a lot to review at once. Or at least one PR with more commits, they don't even all have to build, since the merge queue only squash merges. The latest AGENTS.md has some guidance for this, if you're generating this you can ask your agent something like "please pull the latest upstream AGENTS.md and apply it to the PR". -
I'm not sure we need ATE_REFRESH_CREDENTIAL or the client cert path yet, can we start with the ServiceAccount token default (shared or per-user via --service-account) + client-go's expiry handling, and add the rest when we have a case for it?
We should try hard to not require non-portable flags / dedicated environment variables to get a working client.
nit: please note AI use in the PR description per the PR template
(either way, so far this has been a useful quality signal WRT agents that are not respecting AGENTS.md / docs or humans that are ignoring the PR template ...)
| // mintServiceAccountToken and newCredentialCache are replaced in tests. | ||
| mintServiceAccountToken = ateclient.MintServiceAccountToken | ||
| newCredentialCache = credplugin.DefaultCache |
There was a problem hiding this comment.
🤖 should-fix 🟡 – AGENTS.md asks us not to add package globals just so tests can swap behavior. The test already sets HOME, so the cache override isn't needed, and the mint could go to an httptest server that the test kubeconfig points at instead of a.invalid.
There was a problem hiding this comment.
... please and thank you.
| grpc.WithTransportCredentials(credentials.NewTLS(tlsCfg)), | ||
| grpc.WithStatsHandler(otelgrpc.NewClientHandler()), | ||
| grpc.WithPerRPCCredentials(creds), | ||
| grpc.WithChainUnaryInterceptor(retryUnauthenticated(creds)), |
There was a problem hiding this comment.
🤖 nit 🟢 – The retry is chained outside the trace interceptor and the caller's logging interceptor, so a retried RPC gets two root spans, --trace prints the rejected attempt's trace ID, and -v logs the first attempt as failed. Chaining the retry after opts.GRPCOptions would keep both attempts under one span and one log line.
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return &Cache{Dir: filepath.Join(home, ".ate", "cache", "credentials"), Now: time.Now}, nil |
There was a problem hiding this comment.
🤖 nit 🟢 – os.UserCacheDir() instead of a new ~/.ate dotdir? It follows XDG_CACHE_HOME on Linux.
There was a problem hiding this comment.
This is something we need to nail down early. Probably worth some docs coverage.
| } | ||
|
|
||
| // PrincipalKind is the authentication method that established a principal. | ||
| enum PrincipalKind { |
There was a problem hiding this comment.
cc Julian Gutierrez Oschmann (@juli4n) Tim Hockin (@thockin) , we were discussing only using strings instead of proto enums for compat reasons. We should decide on that ASAP #1378
Part of #1464
Support ExecCredential
kubectl ateauthentication.--credential-plugin(or$ATE_CREDENTIAL_PLUGIN) names a kubectl credential plugin: a path to an executable, or a command line. kubectl ate passes the plugin its--kubeconfigand--contextas$ATE_KUBECONFIGand$ATE_KUBE_CONTEXT.internal/ateclientcaches the plugin's credential until it expires and retries an RPC once onUNAUTHENTICATED. If the plugin returns the rejected token again from its own cache, the client reruns it withATE_REFRESH_CREDENTIAL=1. The default ServiceAccount token now refreshes too, which fixes [P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated #644.kubectl ate credentialmints and disk-caches ServiceAccount tokens (the sharedate-client, or one named with--service-account) and wraps any command that prints a JWT.kubectl ate whoamicalls a newAuthentication.WhoAmIRPC so users can check which principal ate-api sees.Here's how each authentication option looks after this change:
--token-file: you supply the token yourself (Already existed before this change)kubectl ate sends whatever token the file holds. It re-reads the file on every RPC, so a rotated file is picked up automatically, but it never runs anything to get a new token.
--credential-plugin: kubectl ate runs a command that produces the credential (What we're introducing here)The command must print an ExecCredential (v1) as JSON. kubectl ate caches the credential until its expirationTimestamp, and runs the command again when the credential expires or ate-api rejects it.
Rules that apply to both
Reviewer notes:
WhoAmIis a separate service because apitool requires everyControlmethod to name a resource. The exec stanza is fixed atclient.authentication.k8s.io/v1, so plugins that only answer v1beta1 fail. Passing both--token-fileand--credential-pluginis an error.Testing: unit tests; e2e checks WhoAmI with a token file and with a credential plugin (
catof an ExecCredential).Fixes #644